WO01 95633 



Publication Title: 

VIDEO SIZE CONVERSION AND TRANSCODING FROM MPEG-2 TO MPEG-4 



Abstract: 

Abstract of WO01 95633 

A transcoder architecture that provides the lowest possible complexity with a 
small error, e.g., for converting an MPEG-2 bitstream into an MPEG-4 bitstream. 
The transcoder reads header information (304) from an input bitstream and 
provides a corresponding header in the new format for the output bitstream. In 
one embodiment (Fig. 3), a low complexity front-to-back transcoder (with B 
frames disabled) avoids the need for motion compensation processing. In 
another embodiment (Fig. 4), a transcoder architecture that minimizes drift error 
(with B frames enabled) is provided. In another embodiment (Fig. 5), a size 
transcoder (with B frames enabled) is provided, e.g., to convert a bitstream of 
ITU-R 601 interlaced video coding with MPEG-2 MP@ ML into a simple profile 
MPEG-4 bitstream which contains SIF progressive video suitable for a streaming 
video application. For spatial downscaling of field-mode DCT blocks, vertical and 
horizontal downscalling tehniques are combined to use sparse matrixes to 
reduce computations. 

Data supplied from the esp@cenet database - Worldwide 



Courtesy of http://v3.espacenet.com 



This Patent PDF Generated by Patent Fetcher(TM), a service of Stroke of Color, Inc. 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 

International Bureau 

(43) International Publication Date 
13 December 2001 (13.12.2001) 




PCT 



(10) International Publication Number 

WO 01/95633 A2 



(51) International Patent Classification 7 : H04N 7/26 

(21) International Application Number: PCT/US01/4081 1 

(22) International Filing Date: 25 May 2001 (25.05.2001) 

(25) Filing Language: English 

(26) Publication Language: English 



(30) Priority Data: 

09/590,928 



9 June 2000 (09.06.2000) US 



(71) Applicant (for all designated States except US): GEN- 
ERAL INSTRUMENT CORPORATION [US/US] , 101 
Tournament Drive, Horsham, PA 19044 (US). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): PANUSOPONE, 
Krit [TH/US]; 9656 Carroll Canyon Road #F4, San Diego, 
CA 92126 (US). CHEN, Xuemin [US/US]; 8560 Foxcroft 
Place, San Diego, CA 92129 (US). 

(74) Agent: LIPSITZ, Barry, R.; Law Offices of Barry R. Lip- 
sitz, Building No. 8, 755 Main Street, Monroe, CT 06468 
(US). 



(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EE, ES, FI, GB, GD, GE, GH, GM, 
HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, 
LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, MX, 
MZ, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, 
TJ, TM, TR, TT, TZ, UA, UG, US, UZ, VN, YU, ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, 
IT, LU, MC, NL, PT, SE, TR), OAPI patent (BF, BJ, CF, 
CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— without international search report and to be republished 
upon receipt of that report 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



(54) Title: VIDEO SIZE CONVERSION AND TRANSCODING FROM MPEG-2 TO MPEG-4 



-300 



,304 



r 308 



HEADER 


MPEG-2 


LOOK-UP 


MPEG-4 


DECODING 


HEADER 


TABLE 


HEADER 



310 



320 



n 



330 N 



^312 



MPEG-2 




VARIABLE 
LENGTH 
DECODING 




INVERSE 




INVERSE 


BfTSTREAM 




SCAN 




QUANTIZATION 



< 

IT) 

0\ 



O 



340 > 



350 > 



QUANTIZATION 



360 x 



AC/DC 
PREDICTION 



370 



SCAN/ 
RUN- LENGTH 
CODING 



IN- 



VARIABLE 
LENGTH 
ENCODING 



MPEG-4 
^BfTSTREAM 



(57) Abstract: A transcoder architecture that provides the lowest possible complexity with a small error, e.g., for converting an 
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VIDEO SIZE CONVERSION AND TRANSCODING PROM MPEG- 2 TO 

MPEG-4 



BACKGROUND OP THE INVENTION 



The present invention relates to compression of 
5 multimedia data and, in particular, to a video 

transcoder that allows a generic MPEG-4 decoder to 
decode MPEG-2 bitstreams. Temporal and spatial size 
conversion (downscaling) are also provided. 

The following acronyms and terms are used: 
10 CBP - Coded Block Pattern 

DCT - Discrete Cosine Transform 

DTV - Digital Television 

DVD - Digital Video Disc 

HDTV - High Definition Television 
15 FLC - Fixed Length Coding 

IP - Internet Protocol 

MB - Macroblock 

ME - Motion Estimation 

ML - Main Level 
20 MP - Main Profile 

MPS - MPEG-2 Program Stream 

MTS - MPEG-2 Transport Stream 

MV - Motion Vector 

QP - quantization parameter 
25 PMV - Prediction Motion Vector 

RTP - Real-Time Transport Protocol (RFC 1889) 

SDTV - Standard Definition Television 
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SIF - Standard Intermediate Format 

SVCD - Super Video Compact Disc 

VLC - Variable Length Coding 

VLD - Variable Length Decoding 
5 VOP - Video Object Plane 

MPEG-4, the multimedia coding standard, provides a 
rich functionality to support various applications, 
including Internet applications such as streaming 
media, advertising, interactive gaming, virtual 
10 traveling, etc. Streaming video over the Internet 

(multicast) , which is expected to be among the most 
popular application for the Internet, is also well- 
suited for use with the MPEG-4 visual standard (ISO/IEC 
144 96-2 Final Draft of International Standard (MPEG-4) , 
15 "Information Technology - Generic coding of audio- 

visual objects, Part 2: visual," Dec. 1998). 

MPEG-4 visual handles both synthetic and natural- 
video, and accommodates several visual object types, 
such as video, face, and mesh objects. MPEG-4 visual 
20 also allows coding of an arbitrarily shaped object so 

that multiple objects can be shown or manipulated in a 
scene as desired by a user. Moreover, MPEG-4 visual is 
very flexible in terms of coding and display 
configurations by including enhanced features such as 
25 multiple auxiliary (alpha) planes, variable frame rate, 

and geometrical transformations (sprites) . 

However, the majority of the video material (e.g., 
movies, sporting vents, concerts, and the like) which 
is expected to be the target of streaming video is 
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already compressed by the MPEG- 2 system and stored on 
storage media such as DVDs, computer memories (e.g., 
server hard disks), and the like. The MPEG-2 System 
specification (ISO/IEC 13818-2 International Standard 
5 (MPEG-2) , "Information Technology - Generic coding of 

Moving Pictures and Associated Audio: Part 2 - Video," 
1995) defines two system stream formats: the MPEG-2 
Transport Stream (MTS) and the MPEG-2 Program Stream 
(MPS) . The MTS is tailored for communicating or 

10 storing one or more programs of MPEG-2 compressed data 

and also other data in relatively error-prone 
environments. One typical application of MTS is DTV. 
The MPS is tailored for relatively error-free 
environments. The popular applications include DVD and 

15 SVCD. 

Attempts to address this issue have been 
unsatisfactory to date. For example, the MPEG-4 studio 
profile (O. Sunohara and Y. Yagasaki, "The draft of 
MPEG-4 Studio Profile Amendment Working Draft 2.0," 

20 ISO/IEC JTC1/SC29/WG11 MPEG99/5135, Oct. 1999) has 

proposed a MPEG-2 to MPEG-4 transcoder, but that 
process is not applicable to the other MPEG-4 version 1 
profiles, which include the Natural Visual profiles 
(Simple, Simple Scaleable, Core, Main, N-Bit) , 

25 Synthetic Visual profiles (Scaleable Texture, Simple 

Face Animation) , and Synthetic/Natural Hybrid Visual 
(Hybrid, Basic Animated Texture) . The studio profile 
is not applicable to the Main Profile of MPEG-4 version 
1 since it modifies the syntax, and the decoder process 
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is incompatible with the rest of the MPEG- 4 version 1 
profiles. 

The MPEG standards designate several sets of 
constrained parameters using a two-dimensional ranking 
5 order. One of the dimensions, called the "profile" 

series, specifies the coding features supported. The 
other dimension, called "level", specifies the picture 
resolutions, bit rates, and so forth, that can be 
accommodated . 

.10 For MPEG-2, the Main Profile at Main Level, or 

MP@ML, supports a 4:2:0 color subsampling ratio, and I, 
P and B pictures. The Simple Profile is similar to the 
Main Profile but has no B-pictures. The Main Level is 
defined for ITU-R 601 video, while the Simple Level is 
15 defined for SIF video. 

Similarly, for MPEG-4, the Simple Profile contains 
SIF progressive video (and has no B-VOPs or interlaced 
video) . The Main Profile allows B-VOPs and interlaced 
video . 

20 Accordingly, it would be desirable to achieve 

interoperability among different types of end- systems 
by the use of MPEG-2 video to MPEG-4 video transcoding 
and/or MPEG-4 -video to MPEG- 2 -video transcoding. The 
different types of end-systems that should be 

25 accommodated include: 

Transmitting Interworking Unit (TIU) : Receives 
MPEG-2 video from a native MTS (or MPS) system and 
transcodes to MPEG-4 video and distributes over packet 
networks using a native RTP-based. system layer (such as 
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an IP-based internetwork) . Examples include a real- 
time encoder, a MTS satellite link to Internet, and a 
video server with MPS-encoded source material. 

Receiving Interworking Unit (RIU) : Receives MPEG-4 
5 video in real time from an RTP-based network and then 

transcodes to MPEG-2 video (if possible) and forwards 
to a native MTS (or MPS) environment. Examples include 
an Internet -based video server to MTS -based cable 
distribution plant. 
10 Transmitting Internet End-System (TIES) : Transmits 

MPEG-2 or MPEG-4 video generated or stored within the 
Internet end-system itself, or received from internet- 
based computer networks. Examples include a video 
server. 

15 Receiving Internet End-System (RIES) : Receives 

MPEG-2 or MPEG-4 video over an RTP-based internet for 
consumption at the Internet end- system or forwarding to 
a traditional computer network. Examples include a 
desktop PC or workstation viewing a training video. 

20 It would be desirable to determine similarities 

and differences between MPEG-2 and MPEG-4 systems, and 
provide transcoder architectures which yield a low 
complexity and small error. 

The transcoder architectures should be provided 

25 for systems where B-frames are enabled (e.g., main 

profile) , as well as a simplified architecture for when 
B-frames are not used (simple profile) . 

Format (MPEG-2 to MPEG-4) and/or size transcoding 
should be provided. 
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It would also be desirable to provide an efficient 
mapping from the MPEG-2 to MPEG-4 syntax, including a 
mapping of headers . 

The system should include size transcoding, 
5 including spatial and temporal transcoding. 

The system should allow size conversion at the 
input bitstream or output bitstream of a transcoder. 

The size transcoder should convert a bitstream of 
ITU-R 601 interlaced video coded with MPEG-2 MP@ML into 
10 a simple profile MPEG-4 bitstream which contains SIF 

progressive video suitable, e.g., for a streaming video 
application. 

The system should provide an output bitstream that 
can fit in the practical bandwidth for a streaming 
15 video application (e.g., less than 1 Mbps) . 

The present invention provides a system having the 
above and other advantages. 
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SUMMARY OF THE INVENTION 

The invention relates to format transcoding (MPEG- 
2 to MPEG- 4) and size (spatial and temporal) 
transcoding . 

5 A proposed transcoder includes size conversion, 

although these parameters can be transcoded either at 
the input bitstream or the output bit stream. However, 
it is more efficient to include all kinds of 
transcoding into the product version of a transcoder to 
10 reduce the complexity since the transcoders share 

processing elements with each other (such as a 
bitstream reader) . 

The invention addresses the most important 
requirements for a transcoder, e.g., the complexity of 
15 the system and the loss generated by the process. 

In one embodiment, a proposed front- to-back 
transcoder architecture reduces complexity because 
there is no need to perform motion compensation. 

In a particular embodiment, the transcoder can use 
20 variable 5-bit QP representation, and eliminates AC/DC 

prediction and the nonlinear DC scaler. 

The invention is alternatively useful for rate 
control and resizing. 

A particular method for transcoding a pre- 
25 compressed input bitstream that is provided in a first 

video coding format includes the steps of: recovering 
header information of the input bitstream; providing 
corresponding header information in a second, different 
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video coding format; partially decompressing the input 
bit stream to provide partially decompressed data; and 
re -compressing the partially decompressed data in 
accordance with the header information in the second 
5 format to provide the output bitstream. 

A method for performing 2:1 downscaling on video 
data includes the steps of: forming at least one input 
matrix of NxN (e.g., N=16) Discrete Cosine Transform 
(DCT) coefficients from the video data by combining 

10 four N/2xN/2 field-mode DCT blocks; performing vertical 

downsampling and de- interlacing to the input matrix to 
obtain two N/2xN/2 frame-mode DCT blocks; forming an 
NxN/2 input matrix from the two frame-mode DCT blocks; 
and performing horizontal downsampling to the NxN/2 

15 matrix to obtain one N/2xN/2 frame-mode DCT block. 

Preferably, the vertical and horizontal 
downsampling use respective sparse downsampling 
matrixes. In particular, a vertical downsampling 
matrix of 0.5 [I 8 l 8 ] may be used, where I 8 is an 8x8 

20 identity matrix. This is essentially vertical pixel 

averaging. A horizontal downsampling matrix composed 
of odd "O" and even "E" matrices may be used. 

Corresponding apparatuses are also presented. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 illustrates an MPEG-2 video decoder. 

FIG. 2 illustrates an MPEG-4 video decoder without 
any scalability feature. 

FIG. 3 illustrates a low complexity front- to-back 
transcoder (with B frames disabled) in accordance with 
the invention. 

FIG. 4 illustrates a transcoder architecture that 
minimizes drift error (with B frames enabled) in 
accordance with the invention. 

FIG. 5 illustrates a size transcoder in accordance 
with the invention. 

FIG. 6 illustrates downsampling of four field mode 
DCT blocks to one frame mode DCT block in accordance 
with the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

The invention relates to format transcoding (MPEG- 
2 to MPEG-4) and size (spatial and temporal) 
transcoding. 

5 The invention provides bit rate transcoding to 

convert a pre -compressed bitstream into another 
compressed bitstream at a different bit rate. Bit rate 
transcoding is impqrtant, e.g., for streaming video 
applications because the network bandwidth is not 

10 constant and, sometimes, a video server needs to reduce 

the bit rate to cope with the network traffic demand. 
A cascaded-based transcoder which re-uses MVs from the 
input bitstream and, hence, eliminates motion 
estimation (ME) , is among the most efficient of the bit 

15 rate transcoders. The cascaded-based transcoder 

decodes the input bitstream to obtain the MV and form 
the reference frame. It then encodes this information 
with a rate control mechanism to generate an output 
bitstream at the desired bit rate. 

20 Spatial resolution transcoding becomes a big issue 

with the co- existence of HDTV and SDTV in the near 
future. It is also very beneficial for the streaming 
video application since it is likely that the Internet 
bandwidth is not going to be large enough for broadcast 

25 quality video. Hence, downsampling of the broadcast 

quality bitstream into a bitstream with a manageable 
resolution is appealing. Spatial resolution 
transcoding usually performs in the compressed (DCT) 
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domain since it drastically reduces the complexity of 
the system. The process of downsampling in the 
compressed domain involves the processing of two 
parameters, namely DCT coefficients and MVs . A 
5 downsampling filter and its fast algorithm is suggested 

to perform DCT coefficient downsampling. MV resampling 
is used to find the MV of the downsampled video. In 
the real product, to avoid drift, the residual of the 
motion compensation should be re- transformed instead of 
10 approximating the DCT coefficients from the input 

bitstream. 

2. High level comparison 

Structure-wise, MPEG-2 and MPEG-4 employ a similar 
video compression algorithm. Fundamentally, both 
15 standards adopt motion prediction to exploit temporal 

correlation and quantization in the DCT domain to use 
spatial correlation within a frame. This section 
describes the structure of the MPEG- 2 and MPEG-4 
decoders at a high level, and then notes differences 
20 between the two standards. 

2.1 MPEG -2 

FIG. 1' shows the simplified video decoding process 
of MPEG-2. In the decoder 100, coded video data is 
provided to a variable length decoding function 110 to 
25 provide the one-dimensional data QFS [n] , where n is a 

coefficient index in the range of 0-63. At the inverse 
scan function 120, QFS [n] is converted into a two- 
dimensional array of coefficients denoted by QF [v] [u] , 
where the array indexes u and v both lie in the range 0 
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to 7 . An inverse quantisation function 130 applies the 
appropriate inverse quantisation arithmetic to give the 
final reconstructed, frequency- domain DCT coefficients, 
F[v] [u] . An inverse DCT function 140 produces the 
5 pixel (spatial) domain values f [y] [x] . A motion 

compensation function 150 is responsive to a frame 
store memory 160 and the values f [y] [x] for producing 
the decoded pixels (pels) d[y] [x] , where y and x are 
Cartesian coordinates in the pixel domain. 

10 MPEG-2 operates on a macroblock level for motion 

compensation, a block level for the DCT transformation, 
and the coefficient level for run-length and lossless 
coding. Moreover, MPEG-2 allows three types of 
pictures, namely I-, P- and B- pictures. Allowed 

15 motion prediction modes (forward, backward, bi- 

directional) are specified for the P- and B- pictures. 
MPEG-2 uses interlaced coding tools to handle 
interlaced sources more efficiently. 
2.2 MPEG-4 

20 FIG. 2 shows the MPEG-4 video decoding process 

without any scalability features. 

At the decoder 200, data from a channel is output 
from a demux 210. .A coded bit stream of shape data is 
provided to a switch 215, along with the MPEG-4 term 

25 video_object_layer_shape (which indicates, e.g., 

whether or not the current image is rectangular, binary 
only, or grayscale) . If video__object_layer_shape is 
equal to "00" then no binary shape decoding is 
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required. Otherwise, binary shape decoding is carried 
out . 

If binary shape decoding is performed, a shape 
decoding function 220 receives the previous 
5 reconstructed VOP 230 (which may be stored in a 

memory) , and provides a shape-decoded output to a 
motion compensation function 240. The motion 
compensation function 240 receives an output from a 
motion decoding function 235 , which, in turn, receives 

10 a motion coded bit stream from the demux 210. The 

motion compensation function 240 also receives the 
previous reconstructed VOP 230 to provide an output to 
a VOP reconstruction function 245. 

The VOP reconstruction function 245 also receives 

15 data from a texture decoding function 250 which, in 

turn, receives a texture coded bit stream from the 
demux 210, in addition to an output from the shape 
decoding function 220. The texture decoding function 
250 includes a variable length decoding function 255, 

20 an inverse scan function 26 0, an inverse DC and AC 

prediction function 270, an inverse quantization 
function 280 and an Inverse DCT (IDCT) function 290. 

Compared to MPEG-2, several new tools are adopted 
in MPEG-4 to add features and interactivity, e.g., 

25 sprite coding, shape coding, still texture coding, 

scalability, and error resilience. Moreover, motion 
compensation and texture coding tools in MPEG-4, which 
are similar to MPEG-2 video coding, are modified to 
improve the coding efficiency, e.g., coding tools such 
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as direct mode motion compensation, unrestricted motion 
compensation, and advanced prediction. 

In particular, direct mode motion compensation is 
used for B-VOPs. Specifically, it uses direct bi- 
5 directional motion compensation derived by employing I- 

or P-VOP macroblock MVs and scaling them to derive 
forward and backward MVs for macroblocks in B-VOP. 
Only one delta MV is allowed per macroblock. The 
actual MV is calculated from the delta vector and the 
10 scaled MV from its co-located macroblock. 

Unrestricted motion compensation allows one or 
four MVs per macroblock. The four MV mode is only 
possible in B-VOPs with the use of direct mode. Note 
that the MV for a chrominance macroblock is the average 
15 of four MVs from its associated luminance macroblock. 

Furthermore, unrestricted motion compensation allows an 
MV to point out of the reference frame (the out -of - 
bound texture is padded from the edge pixel) . 

Advanced prediction defines the prediction method 
20 for MV and DCT coefficients. A MV predictor is set 

according to the median value of its three neighbors 7 
MVs. Prediction of the intra DCT coefficient follows 
the intra AC/DC prediction procedure (Graham's rule) . 
3 . Transcoder architecture 
25 FIG. 3 illustrates a low complexity front- to-back 

transcoder in accordance with the invention, with B 
frames disabled. 

Similarities between the structures of MPEG- 2 and 
MPEG-4 allow a low complexity (front -to-back) 
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transcoder. Instead of completely decoding an MPEG-2 
bit stream to the spatial (pixel) domain level, the 
front-to-back transcoder 300 uses DCT coefficients and 
MVs to generate an MPEG-4 bitstream without actually 
5 performing a motion estimation process. A trade-off is 

that this architecture may cause a drift in the 
reconstructed frame, and does not allow bit rate 
control. However, the drift problem is small since 
most of the difference between the MPEG-2 and MPEG-4 

10 decoders lies in the lossless coding part. 

The transcoder 3 00 comprises a cascade of a MPEG-2 
bitstream reader (decoder) (310-330) and a MPEG-4 
header and texture coder (encoder) (340-370) , along 
with a header decoding function 304, a look-up table 

15 308, and a communication path 312. The transcoder 300 

reads an input MPEG-2 bitstream, performs a variable 
length decoding (VLD) at a function 310 on DCT 
coefficients and MV residual, and then follows MPEG-2 
logic to find DCT coefficients and/or MVs of every 

20 block in the frame. 

The header decoding function 304 decodes the MEPG- 
2 headers and provides them to a look-up table (or 
analogous function) 3 08, which uses the tables detailed 
below to obtain corresponding MPEG-4 headers. 

25 With the information of the headers, DCT 

coefficients and/or MV, the transcoder 300 encodes this 
information into the MPEG-4 f ormat . Note that the 
reference frame is not needed in this architecture. 
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The transcoder 300 reads the MPEG- 4 header from 
the input bitstream and writes the corresponding MPEG-4 
header in its place in an output bitstream. 

After processing at the VLD 310,- the data is 
5 provided to an inverse scan function 320, and an 

inverse quantisation function 330. Next, using the 
MPEG-4 header information provided via the path 312, 
the decoded, DCT coefficient data is processed at a 
MPEG-4 header and texture coder that includes a 
10 quantisation function 340, and an AC/DC prediction 

function 350 for differentially encoding the quantised 
DCT coefficients. In particular, the AC /DC prediction 
process generates a residual of DC and AC DCT 
coefficients in an intra MB by subtracting the DC 
15 coefficient and either the first row or first column of 

the AC coefficients. The predictor is adaptively 
selected. Note that the AC/DC prediction function 350 
may not need the MPEG-4 header information. 

Subsequently, a scan/run- length coding function 
20 360 and a variable length encoding function 3 70 provide 

the MPEG-4 bitstream. 

FIG. 4 illustrates a transcoder architecture that 
minimizes drift error in accordance with the invention, 
with B frames enabled. 
25 Like-numbered elements correspond to one another 

in the figures. 

To counter the problems of drift in the 
reconstructed frame, and the lack of bit rate control, 
a more complex, architecture such as the transcoder 4 00, 
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which is an extension of the transcoder 300 of FIG. 3, 
can be used. This architecture actually computes the 
DCT coefficient of the texture/residual data, hence 
motion compensation is required. Since the encoder of 
5 this transcoder includes a decoding process, the drift 

error can be minimized. 

Moreover, the transcoder 400 can be used to 
transcode bitstreams with B-frames since MPEG-4 does 
not allow intra mode for B-frames. The transcoder 400 

10 treats a block in intra mode in a B-frame (in MPEG-2) 

as a block with a zero MV in inter mode (in MPEG-4) . It 
can be either a zero residual MV (PMV) or zero MV 
(which may yield a non-zero MV code) since the MV is 
predictive coded against the PMV. 

15 In particular, the transcoder 400 includes a 

variable length decoding function 405 that provides MV 
residue data to a MV decoder 425, and that provides DCT 
coefficient data to the inverse scan function 320. The 
DCT data is processed by the inverse quantisation 

20 function 330 and an inverse DCT function 420 to obtain 

pixel domain data. Intra-coded pixel data is provided 
via a path 422 to a buffer, while inter-coded pixel 
data is provided to an adder 435 via a path 424. 

The pixel (difference) data on path 424 is added 

25 to reference pixel data from a motion compensation 

function 430 (responsive to the MV decoder 425) to 
provide inter-coded data to the buffer 450 via a path 
448. 
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For re -encoding, e.g., in the MPEG -4 format, the 
buffer 450 either outputs the intra pixel data directly 
to a DCT function 455, or outputs the inter pixel data 
to a subtractor 445, where a difference relative to an 
5 output from a motion compensation function 44 0 

(responsive to the MV decoder 425) is provided to the 
DCT function 455. 

The DCT coefficients are provided from the DCT 
function 455 to the quantisation function 340, and the 
10 quantised DCT data is then provided to the AC/DC (DCT 

coefficient) prediction function 350, where AC and DC 
residuals of the current MB are generated. These 
residuals of DCT coefficients are entropy coded. The 
output data is provided to the scan/ run -length coding 
15 function 360, and the output thereof is provided to the 

variable length encoding function 370 to obtain the 
MPEG-4 compliant bitstream. 

The quantised DCT coefficients are also output 
from the quantisation function 340 to an inverse 
20 quantisation function 495, the output of which is 

provided to an inverse DCT function 490, the output of 
which is summed at an adder 485 with the output of the 
motion compensation function 440. The output of the 
adder 485 is provided to a buffer 480, and subsequently 
•25 to the motion compensation function 440. 

The header decoding function 304 and look-up table 
308 and path 312 operate as discussed in connection 
with FIG. 3 to control the re-encoding to the MPEG-4 
format at functions 340-370. 
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4. Implementation of the Format Transcoder 

This section explains the implementation of the 
format transcoding, e.g., as implemented in FIGs 3 and 
4, discussed above, and FIG. 5, to be discussed later. 
5 Minor implementation details (e.g., systems-related 
details such as the use of time stamps and the like) 
that are not specifically discussed should be apparent 
to those skilled in the art. 

In a particular implementation, the transcoder s of 

10 the present invention can be used- to convert a main- 
profile, main-level (MP@ML) MPEG-2 bitstream into a 
main-profile MPEG-4 bitstream. It is assumed that the 
MPEG-2 bitstream is coded in frame picture structure 
with B-picture coding (no dual prime prediction) . 

15 Generally, the same coding mode which is used in MPEG-2 
coding should be maintained. This mode is likely to be 
optimum in MPEG-4 and hence avoids the complexity of 
the mode decision process. The transparency pattern in 
MPEG-4 is always 1 (one rectangular object with the 

20 same size of VOP in one VOP) . That is, MPEG-4 allows 

an arbitrarily shaped object which is defined by a 
; nonzero transparency pattern. This feature does not 
exist in MPEG-2 so we can safely assume that all 
transparency patterns of the transcoding object is one. 

25 4.1 MPEG-2 bitstream reader 

A transcoder in accordance with the invention 
obtains the bitstream header, DCT coefficients and MVs 
from the MPEG-2 bitstream. This information is mixed 
together in the bitstream. Both MPEG-2 and MPEG-4 
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bitstreams adopt a hierarchical structure consisting of 
several layers. Each layer starts with the header 
following by a multiple of its sublayer. In this 
implementation, as shown in Table 1, the MPEG-2 layer 
has a direct translation into the MPEG-4 layer, except 
the slice layer in MPEG-2, which is not used in MPEG-4. 
DC coefficients and predicted MVs in MPEG-4 are reset 
at the blocks that start the slice. 

However, some MPEG-4 headers are different from 
MPEG-2 headers, and vice versa. Fortunately, the 
restrictions in MPEG-2 and MPEG-2 header information 
are sufficient to specify a MPEG-4 header. Tables 2 
through 6 list MPEG-4 headers and their relation to a 
MPEG-2 header or restriction at each layer. 



Table 1. Relationship between MPEG-2 and MPEG-4 
layers 



MPEG-2 


MPEG-4 


Video Sequence 


Video Object Sequence (VOS) / 
Video Object (VO) 


Sequence Scalable Extension 


Video Object Layer (VOL) 


Group of Picture (GOP) 


Group of Video Object Plane (GOV) 


Picture 


Video Object Plane (VOP) 


Macroblock 


Macroblock 


Table 2. MPEG-4 header and its derivation (VOS and 

VO) 


Header 


Code 


Comment 


Visual_object_sequence_start_code 


00001 BO 


Initiate a visual session 


Profile_andJevel_indication 


00110100 


Main Profile/Level 4 
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Visual_object_sequence_end_code 


00001 B1 


Terminate a visual session 


Visual_object_start_code 


00001 B5 


Initiate a visual object 


ls_visual_object_identifier 


0 


No version identification of priority 
needs to be specified 


Visual_objectJype 


0001 


Video ID 


Video_object_start_code 


000001 ox- 
000001 1X 


Mark a new video object 


Video_signal_type 


Derived 

from 

MPEG-2 


Corresponds to MPEG-2 
sequence_display_extension_id 


Videojormat 


Same as 
MPEG-2 


Corresponds to MPEG-2 
sequence_display_extension_id 


Video_range 


Derived 
from 

MPEG-2 ' 


Corresponds to MPEG-2 
sequence_display_extension_id 


Colour_description 


Same as 
MPEG-2 


Corresponds to MPEG-2 
sequence_display_extension_id 


Colour_primaries 


Same as 
MPEG-2 


Corresponds to MPEG-2 
colour_description 


Transfer_characteristics 


Same as 
MPEG-2 


Corresponds to MPEG-2 
colour_description 


Matrix_coefficients 


Same as 
MPEG-2 


Corresponds to MPEG-2 
colour_description 


Table 3. MPEG-4 header and its derivation (VOL) 


Header 


Code 


Comment 


Video_object_layer_start_code 


0000012X 


Mark a new video object layer 


Random_accessible_vol 


0 


Allow non-intra coded VOP 


Video_object_type_identification 


00000100 


Main object type 


ls_object_ty pe_id e ntifier 


0 


No version identification of priority 
needs to be specified 


Aspect_ratio_info 


Same as 
MPEG-2 


Corresponds to-MPEG-2 
aspect_ratio_information 


Par_width 


Same as 


Corresponds to MPEG-2 



wo 
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MPEG-2 


vertical_size 


Parjieight 


Same as 


Corresponds to MPEG-2 




MPEG-2 


horizontal_size 


Vol_control_parameters 


Same as 


Correponds to MPEG-2 




MPEG-2 


extension_start_code_identifier 






(sequence extension) 


Chromajormat 


Same as 


Corresponds to MPEG-2 




MPEG-2 


chromajormat 


Low_delay 


Same as 


Corresponds to MPEG-2 low_delay 




MPEG-2 




Vbv_parameters 


Recomputed 


Follow MPEG-4VBV spec. 


Video_object_layer_shape 


00 


Rectangular 


Vop_timejncrement_resolution 


Recomputed 


See Table 7 


Fixed_vop_rate 


1 


Indicate that all VOPs are coded at 






a fixed rate 


Fixed_vop_time_increment 


Recomputed 


See Table 7 


Video_object_layer_width 


Same as 


Correpond to display_vertical_size 




MPEG-2 




Video_object_layer_height 


Same as 


Correspond to 




MPEG-2 


display_horizontal_size 


Interlaced 


Same as 


Correspond to 




MPEG-2 


progressive_sequence 


Obmc_disable 


1 


Disable OBMC 


Sprite_enable 


0 


Indicate absence of sprite 


Not_8_bit 


Derived from 


Corresponds to MPEG-2 




MPEG-2 


intra_dc_precision 


Quantjype 


1 


MPEG quantization 


Complexity_estimation_disable 


1 


Disable complexity estimation 






header 


Resync_marker_disable 


1 


Indicate absence of resync_marker 


Data_partitioned 


0 


Disable data partitioning 


Reversible_vlc 


0 


Disable reversible vie 


Scalability 


0 


Indicate that the current layer is 






used as base-layer 
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Table 4 . MPEG-4 header and its derivation (VOP) 



Header 


Code 


Comment 


Vop_starLcode 


000001 B6 


Mark a start of a video object plane 


Vop_coding_type 


Same as 


Corresponds to MPEG-2 




MPEG-2 


picturej»dingjype 


ModuIoJime_base 


Regenerated 


Follow MPEG-4 spec. 


Vopjimejncrement 


Regenerated 


Follow MPEG-4 spec. 


Vop_coded 


1 


Indicate that subsequent data exists 
for the VOP 


Vopjoundingjype 


0 


Set value of rounding control to '0' 


Change_conversion_ratio_disable 


1 


Assume that conv_ratio is T for all 
macroblocks 


Vop_constanLalpha 


0 


Not include 

vop_constant_alpha_value in the 
bitstream 


lntra_dc_vlc_thr 


0 


Use intra DC vie for entire VOP 


Topjieldjirst 


Same as 


Corresponds to MPEG-2 




MPEG-2 


top_field_first 


Alternate j/erticaLscan Jag 


Same as 


Corresponds to MPEG-2 to 




MPEG-2 


alternate_scan 


Vop_quant 


Derived from 


Corresponds to MPEG-2 




MPEG-2 


quantiser_scale_code 


Vopjcodejorward 


Same as 
MPEG-2 


See section 4.3 


Vopjcode^backward 


Same as 
MPEG-2 


See section 4.3 



Table 5. MPEG-4 header and its derivation 
(macroblock and MV) 



Header 


Code 


Comment 


Notcoded 


Derived from MPEG-2 


Corresponds to MPEG-2 
macroblock_addressJncrement 
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uerivBQ Trom ivirtb-z 


Corresponds to MPfcb-z 
macrouiocKjype 


Ap nrpH flan 


n 


uisaDie inira preaiction 


Cbpy 


Derived from MPEG-2 


See section 4.2 




uenvea irom Mrbb-^ 


See section 4.2 


Modb 


Derived from MPEG-2 


Corresponds to macroblockjype 


iviD_type 


Derived from MPEG-2 


Corresponds to macroblockjype 


Cbpb 


Derived from MPEG-2 


See section 4.2 


Dbquant 


Denved from MPEG-2 


See section 4.2 


Horizontal_mv_data 


Derived from MPEG-2 


Conesponds to MPEG-2 
motio n_code [r][s] [0] 


Vertical_mv_data 


Derived from MPEG-2 


Corresponds to MPEG-2 
motion_code[r][s][1] 


Horizontal_mv_residual 


Derived from MPEG-2 


Corresponds to MPEG-2 
motion_residual[r][s][0] 


Vertical_mv_residual 


Derived from MPEG-2 


Corresponds to MPEG-2 
motion_residual[r][s][1] 



Table 6. MPEG-4 header and its derivation (block 
and interlaced information) 



Header 


Code 


Comment 


Dct_dc_size_luminance 


Same as MPEG-2 


Corresponds to MPEG-2 
dct_dc_size_luminance 


Dct_dc_differentia! 


Same as MPEG-2 


Correspond to dct_dc_d iffere ntial 


Dct_dc_size_chrominance 


Same as MPEG-2 


Corresponds to MPEG-2 
dct_dc_size_chrominance 


DCT_coefficient 


Derived from 
MPEG-2 


See section 4.2 


DCTJype 


Same as MPEG-2 


Corresponds to MPEG-2 
DCTJype 


Field_prediction 


Same as MPEG-2 


Corresponds to MPEG-2 
frame jnotionjype 


Forward_top_field_reference 


Same as MPEG-2 


Corresponds to MPEG-2 
motion_vertical_field_select[0][0] 
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Forward_bottom_field_reference 


Same as MPEG-2 


Corresponds to MPEG-2 
motion_verticaljield_select[1][0] 


Backward_top_field_reference 


Same as MPEG-2 


Corresponds to MPEG-2 
motion_vertical_field_select[0][1] 


Backward_bottom_field_reference 


Same as MPEG-2 


Corresponds to MPEG-2 
motion_vertical_field_select[1 ][1 ] 



Table 7. Mapping of frame_r at encode in MPEG-2 to 
vop_time_increment__resolution and 
f ixed_vop_time_increment in MPEG-4. 



Frame_rate_code 


Vop_time_increment_resolution 


Fixed_vopJime_increment 


0001 


24,000 


1001 


0010 


24 




0011 


25 




0100 


30,000 


1001 


0101 


30 




0110 


50 




0111 


60,000 


1001 


1000 


60 





MV data is stored in the macroblock layer. Up to 
5 four MVs are possible for each macroblock. Moreover, a 

MV can be of either field or frame type and have either 
full pixel or half pixel resolution. The MPEG-2 MV 
decoding process is employed to determine motion_code 
(VLC) and motion_residual (FLC) and, hence, delta. 
10 Combined with predictive MV, delta gives the 

field/frame MV. The MV for skipped macroblocks is set 
to zero. 

DCT data is stored in the block layer. It is 
first decoded from the bitstream (VLC) , inverse scanned 
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using either zigzag or alternate scanning pattern, and 
then inverse quantized. The intra DC coefficient is 
determined from dct_dc_dif f erential and the predictor 
(the predictor is reset according to the MPEG- 2 spec) . 
5 DCT coefficients in a skipped macroblock are set to 

zero. 

4 • 2 Texture coding 

A transcoder in accordance with the invention 
reuses DCT coefficients (for inter frame) . The 
10 following guidelines should be used: 

1. q_scale_type = 1 (linear scale) is used in 
MPEG- 2 quantization. 

2. The MPEG quantization method should only be 
used (not H.263) in MPEG-4 quantization to reduce a 

15 mismatch between MPEG- 2 and MPEG-4 reconstructed frame 

(drift) . 

3. A differential value of MPEG-2 QP determines 
dquant in MPEG-4 . Dquant is set to ±2 whenever the 
differential value is greater than ±2. dquant is a 2- 

20 bit code which specifies a change in the quantizer, 

quant, for I- and P-VOPs. 

4. The quantization matrix should be changed 
following the change of matrix in the MPEG-2 bitstream. 

5. The transcoder has the flexibility of 

25 enabling an alternate vertical scanning method (for 

interlaced sequence) at the VOL level. 

6. Intra AC/DC prediction (which involves 
scaling when the QP of the current block is not the 
same as that of the predicted block) should be turned 
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off at a macroblock level to reduce complexity and 
mismatch in AC quantization. 

7. Higher efficiency can be obtained with the 
use of intra_dc_vlc_thr to select the proper VLC table 
5 (AC/DC) for coding of intra DC coefficients, e.g., as a 

function of the quantization parameter (except when 
intra_dc_vlc_thr is either 0 or 7 - these thresholds 
will force the use of the intra DC or AC table 
regardless of the QP) . 

10 8. A skipped macroblock is coded as not_coded 

macroblock (all DCT coefficients are zero) . 

9. Cbpy and cbpc (CBP) are set according to 
code_block_pattern_420 (CBP_420) . Note that there is a 
slight discrepancy between CBP in MPEG-4 and CBP_420 in 

15 MPEG-2 for an intra macroblock. Specifically, when 

CBP_420 is set, it indicates that at least one of the 
DCT coefficients in that block is not zero. CBP 
contains similar information except it does not 
corresponds to a DC coefficient in an intra macroblock 

20 (also depending on intra_dc_vlc_thr) . Hence, it is 

possible that CBP is not zero when CBP_420 is zero in 
an intra macroblock (this case can happen in an I-VOP 
and P-VOP, but not B-VOP) . 

There are three sources of loss in texture coding, 

25 namely QP coding, DC prediction and nonlinear scaler 

for DC quantization. MPEG-4 uses differential coding 
to code a QP. MPEG-2 allows all possible 32 QP values 
at the expense of 5 bits. However, the differential 
value can take up to ±2 (in QP value units) and, hence, 
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a differential value greater than ±2 is loss. This loss 
can be minimized by limiting the QP fluctuation among 
the macroblock in the MPEG-2 rate control algorithm. 
All intra macroblocks perform adaptive DC prediction, 
5 which may take a different prediction from the previous 

macroblock (MPEG-2 DC prediction) thereby causing a 
different DC residual for the quantization. DC 
coefficients of all intra macroblocks in MPEG-4 are 
also quantised in a different manner from MPEG-2 
10 because of the nonlinear scaler. Therefore, quantised 

DC coefficients for MPEG-2 and MPEG-4 coding are likely 
to be different for an intra macroblock. 
4.3 MV coding 

The transcoder encodes MVs into an MPEG-4 format. 
15 However, there is no error involved in transcoding a MV 

from MPEG-2 to MPEG-4 since MV coding is a lossless 
process. The following constraints are imposed on a 
MPEG-4 encoder: 

1. Unrestricted motion compensation mode is 
20 disabled, which means no MV pointing outside the 

boundary of the frame. 

2. Advanced prediction mode is employed. A 
different predictor (a median value) is used in an 
MPEG-4 bitstream, but a MV for 8x8 pels block is not. 

25 That is, advanced prediction mode allows 8x8 MV and 

nonlinear (median filter) predictor. Only a nonlinear 
predictor is used in our format transcoder (we still 
keep a 16x16 MV) . 
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3. Direct mode is not allowed in an MPEG-4 
bitstream, which means there are only four MV types for 
a B-VOP, i.e., 16x16 forward and backward vectors and 
16x8 forward and backward field vectors. 
5 4 . Field motion compensation is applied whenever 

a 16x8 field vector is used (maintain mode) . 

5. A skipped macroblock is coded as not_coded 
macroblock (motion compensation with zero MV) . 

6. Single f_code is allowed in MPEG-4. 

10 Therefore, the larger f_code in MPEG- 2 between the two 
directions (vertical, horizontal) is converted to 
f__code in MPEG-4 based on the following relationship: 
f__code(MPEG-4) = f_code (MPEG-2) -1 . 

7. A padding process is not used since the 
15 texture for the entire reference frame is known. 

8. Field motion compensation is used whenever 
dual prime arithmetic is activated. Vector parities 
(field of the reference and field of the predicting 
frame) are preserved. Field MVs are generated 

20 according to vector [0] [0] [1:0] which is coded in the 
MPEG-2 bitstream. When prediction of the same parity 
is used (e.g., top field to top field, or bottom field 
to bottom field) , both field MVs are vector [0] [0] [1:0] . 
When prediction of the odd parity is used (e.g., top 

25 field to bottom field, or bottom field to top field) , 

the top field MV uses vector [2] [0] [1:0] and the bottom 
field MV uses vector [3] [0] [1:0] . Vector [r] [0] [0:1] for 
r=2,3 can computed as follows: 
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(a) Vector [r] [0] [0] = (vector [0] [0] [0] x 

m [parity_ref ] [parity_pred] //2) + dmvector[0] . 

(b) Vector [r] [0] [1] = (vector [0] [0] [1] x 
m [parity_ref ] [parity_pred] //2) + 

5 e [parity_ref ] [parityjpred] + dmvector[l] . 

Note that (m [parity_ref ] [parity_j>red] and 
e [parity_ref ] [parityjpred] are defined in Table 7-11 
and 7-12, respectively in the MPEG-2 specification 
(ISO/IEC 13818-2) . 
10 Moreover, "r" denotes the order of the MV, e.g., 

first, second, etc. r=0 denotes to the first set of 
MV, and r=l denotes the second set of MV. Dual prime 
prediction uses r=2 and r=3 to identify two extra sets 
of MVs. 

15 "//" denotes integer division with rounding to the 

nearest integer. 

4.4 Coding of intra MB in B-VOP 

Additional conversion is necessary when coding an 
intra MB in a B-frame of a MPEG-2 bitstream (e.g., as 

20 shown in FIG. 4) . MPEG- 4 replaces intra mode with 

direct mode for B-VOP and hence an intra MB in B-frame 
has to be coded differently in the MPEG-4 syntax. 
There are two practical solutions to this problem. 
The first solution employs the architecture 

25 similar to the front- to-back transcoder of FIG. 3 (no 

buffer for the entire reference frame) . MC is 
performed against the previous MB (or previous MB 
without compensating texture residual with the expense 
of the extra memory with the size of one MB) in the 
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same VOP under the assumption that this MB is close 
enough to its reference MB (its uncompensated version) . 
The MV for the intra MB equals the MV of the previous 
MB offset by its MB distance . 
5 The second solution uses the architecture similar 

to the one shown in FIG. 4. It keeps the reference 
frame for all I and P-VOPs. Note that MC has to be 
performed on all P-VOPs in this solution. The MV for 
the intra MB is the same as the predicted MV (median of 

10 its three neighbors) and MC is performed against the 

reference MB pointed by the derived MV. 

5. Video downscaling in the compressed domain 
Generally, video downscaling and size transcoding 
have the same meaning. Downsampling means sub-sampling 

15 with an anti-aliasing (low pass) filter, but 

subsampling and downsampling are used interchangeably 
herein. 

Size transcoding becomes computationally intensive 
when its input and output are in the compressed domain. 

20 A video downscaling process which limits its operations 
in the compressed domain (and, in effect, avoids 
decoding and encoding processes) provides a much 
reduced complexity. However, two new problem arises 
with downscaling in the compressed domain, i.e.., 

25 downsampling of DCT coefficients and MV data. 

Recently, video downscaling algorithms in the 
compressed domain have been discussed, but they do not 
address the complete transcoding between MPEG-2 and 
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MPEG-4, which includes field- to- frame deinterlacing . 
The present invention addresses this problem. 

Subsection 5.1 and 5.2 provide solutions to two 
new problems in the downsampling process. The 
5 implementation of a proposed size transcoder in 

accordance with the invention is described in section 6 
and FIGs 5 and 6. 

5.1 Subsampling of DCT block 

In frame-based video downscaling, it is necessary 

10 to merge four 8x8 DCT blocks into a new 8x8 DCT block 

(specific details involving a field block will be 
described later) . Moreover, the output block should be 
a low pass version of the input blocks. This process 
is carried out in the spatial domain by multiplying the 

15 input matrix with a subsampling matrix (preferably with 

a low pass filter) . Multiplication by a subsampling 
matrix in the spatial domain is equivalent to 
multiplication by DCT coefficients of a subsampling 
matrix in the DCT domain because of the distributive 

20 property of the orthogonal transform. However, the 

number of operations (computations) in the downsampling 
process in the DCT domain for some downsampling filters 
can be as high as the total number of operations of its 
counterpart in the spatial domain. The solution to 

25 this problem is to employ a downsampling matrix which 

is sparse (e.g., a matrix that has relatively few non- 
zero values, e.g., approximately 30% or less). 

A sparse downsampling matrix may be based on the 
orthogonal property between the DCT basis vector and 
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the symmetry structure of the DCT basis vector. One 
approach, discussed in R. Dugad and N. Ahuja, n A Fast 
Scheme For Downsampling And Upsampling In The DCT 
Domain, " International Conference on Image Processing 
5 (ICIP) 99, incorporated herein by reference, takes the 

lower 4x4 DCT coefficients from four processing blocks, 
applies a 4x4 IDCT to each DCT subblock, forms a new 
8x8 pixel block and applies an 8x8 DCT to obtain an 
output block. The downsampling matrix can be pre- 

10 calculated since the downsampling process is fixed. By, 

splitting the 8x8 DCT matrix into left and right 
halves, about half of the downsampling matrix values 
are zero because of the orthogonality between the 
column of the 4x4 IDCT matrix and the row of both left 

15 and right 8x4 DCT matrices. This operation (one 

dimension) can be written mathematically as: 



= T L TiB l + T R 4B 2 



where b is a 8x1 spatial input vector, B is its 
corresponding 8x1 DCT vector, b z and b 2 are subsampled 

20 4x1 vectors, B x and B 2 are lower 4x1 DCT vectors, T is 

the 8x8 DCT transform matrix, T 4 is the 4x4 DCT 
transform matrix, T L and T R are left and right half of 
T. The superscript "t" denotes a matrix transpose. 
Dugad' s algorithm also employs the symmetry property of 

25 the DCT basis vector to reduce the complexity of the 

downsampling process. T L T\ an( } T R T\ are identical in 





V 




~tIb x 


B = Tb = T 




= [T L I T R ] 










TiB 2 
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terms of magnitude ( T L Tj(iJ) = (-iy+J T R 4(iJ) 9 0<i<7,0< j <:3 ) since 
odd rows of T are anti- symmetric and even rows of T are 
symmetric. "i" is a matrix row index, and "j" is a 

matrix column index. Hence, both T L T\ an d T r t\ can be 
5 calculated based on the same components, i.e., a 

symmetrical part, E, (index which i+j is even) and an 
anti -symmetrical part, O, (index which i+j is odd) 

( T L T 4 =s+0 and T R Tl =e-0) . This arrangement effectively 
reduces the number of multiplications by a factor of 
10 two when the downsampling process is done as: 

Implementation of Dugad's method to convert four 
field blocks into one frame block is. not as simple. An 
15 extension of the downsampling process in this scenario 

(one dimension) can be written as: 

B = T(S T TlB T + S B TlB B ) 
where B T and B B are the lower 4x1 field vectors, S T and 
S B are DCT values of an 8x4 deinterlacing matrix 
20 corresponding to its top, s Tt and bottom, s B , field 

block, respectively. Elements of s Tt s T (i,j)=lL if 
(j=2i, 0<i<3; and s T (i,j)=0 otherwise. Elements of s B , 
s B (i,j)=l if (j=2i+l, 0<i<3; and s B (i,j;=0 otherwise. 
This is a modification of Dugad's algorithm for 
25 downsampling and deinterlacing in accordance with the 

present invention. 
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10 



15 



20 



The operations of downscaling and the 
deinterlacing process are more complex since S and T 
are not orthogonal to each other and, hence, the 
downsampling matrix is not sparse. C. Yim and M . A. 
Isnardi, "An Efficient Method For DCT-Domain Image 
Resizing With Mixed Fie Id/ Frame -Mode Macroblocks, " IEEE 
Trans . Circ . and Syst . For Video Technol . , vol . 9 . 
pp. 696-700, Aug. 1999, incorporated herein by- 
reference, propose an efficient method for downsampling 
a field block. A low pass filter is integrated into 
the deinterlacing matrix in such a way that the 
downsampling matrix (S=0.5[I 8 I 8 ] ) is sparse. 

I 8 denotes an 8x8 identity matrix, and [I 8 I 8 ] 
denotes a 16x8 matrix that comprises a concatenation of 
the two identity matrixes. The identity matrix, of 
course, has all ones on the diagonal and all zeroes 
elsewhere. 

The method starts with four 8x8 IDCT field blocks, 
then applies the downsampling matrix, S, and performs 
an 8x8 DCT to obtain the output block. Note that an 
8x8 IDCT is used in this method instead of a 4x4 IDCT. 
This operation can be shown mathematically (in one 
dimension) as : 

'r o 



D = TS 



o r 



= 1 ~T\I : I] P 0 



-\[h i /.] 
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5.2 Subsampling of MV data 
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ME is the bottleneck of the entire video encoding 
process. It is hence desirable to estimate a MV of the 
resized MB by using MVs of four original MBs without 
actually performing ME (assuming that all MBs are coded 
5 in inter mode) . Note that, if an MPEG-2 bitstream is 

assumed, subsampling of MV data takes MVs of four MBs 
since each MB has one input (only an MPEG-4 bitstream 
can have a MV for every block) . The simplest solution 
is to average four MVs together to obtain the new MV 

10 but it gives a poor estimate when those four MVs are 

different. B. Shen, I.K. Sethi and B. Vasudev, 
"Adaptive Motion-Vector Resampling For Compressed Video 
Downscaling, " IEEE Trans. Circ. and Syst . For Video 
Technol., vol. 9, pp. 929-936, Sep. 1999, show that a 

15 better result can be obtained by giving more weight to 

the worst predicted MV. A matching accuracy, A, of 
each MV is indicated by the number of nonzero AC 
coefficients in that MB. By using the Shen et al . 
technique, the new MV for the downscaled MB can be 

2 0 computed as: 

. 4 

MV'=--^- A 

/=1 

M.R. Hashemi, L. Winger and S. Panchanathan, 
"Compressed Domain Motion Vector Resampling For 
Downscaling Of MPEG Video, 11 ICIP 99, propose a 
25 nonlinear method to estimate the MV of the resized MB. 

Similar to the algorithm in Shen et al . , Hashemi's 
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technique uses spatial activity of the processing MBs 
to estimate the new MV. A heuristic measurement, 
called Maximum Average Correlation (MAC) is employed in 
Hashemi's method to identify one of the four original 
MVs to be the output MV. By using the MAC, the new^MV 
for the downscaled MB can be computed as : : 



where p is the spatial correlation and is set to 0.85, 
and di is the Euclidean distance between the ith input 
MV (MVi) and the output MV. 

6, Implementation of the size transcoder 
FIG. 5 illustrates a size transcoder in accordance 
with the invention. B frames may be present in the 
input bitstream, but are discarded by the transcoder 
and therefore do not appear in the output bitstream. 

In the transcoder 500, a MV scaling function 510, 
DCT scaling function 520, and spatial scaling function 
540 are added. Switches 530 and 535 are coordinated so 
that, in a first setting, an output of the DCT function 
455 is routed into the quantisation function 340, and 
the switch 53 5 is closed to enable an output of the 
spatial scaling function 540 to be input to the adder 
445. In a second setting of the switches 53 0 and 535, 
an output of the DCT scaling function 520 is routed 
into the quantisation function 340, and the switch 535 
is open. 



4 




/=1 
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The transcoder 500 converts an MPEG-2 bitstream 
into an MPEG-4 bitstream which corresponds to a smaller 
size video, e.g., from ITU-R 601 (720x480) to SIF 
(352x240) . 

5 To achieve a bandwidth requirement for the MPEG-4 

bitstream, the transcoder 500 subsamples the video by 
two in both the horizontal and vertical directions (at 
the spatial scaling function 540) and skips all B- 
frames (at temporal scaling functions 545 and 546) , 

10 thereby reducing the temporal resolution accordingly. 

Note that the temporal scaling function 546 could 
alternatively be provided after the DCT scaling 
function 520. Skipping of B-frames before performing 
downscaling reduces complexity. 

15 Moreover, a low pass filter (which can be provided 

in the spatial scaling function 540) prior to 
subsampling should result in improves image quality. 

The invention can be extended to include other 
downsampling factors, and B-VOPs, with minor 

20 modifications. Specifically, changes in MV downscaling 

and mode decision are made. MV downscaling for B-VOP 
is a direct extension of what was discussed to include 
the backward MV. The mode decision for B-VOP can be 
handled in a similar way as in the P-VOP (e.g., by 

25 converting uni -directional MV into bi-directional MV as 

in converting intra MB into inter MB in a P-VOP) . 

Below, we discuss six problems that are addressed 
by the size transcoder 500. We also assume that the 
input . video is 704x480 pixel resolution, and coded with 
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an MP@ML MPEG -2 encoder, and the desired output is 
simple profile MPEG-4 bitstream which contains SIF 
progressive video (with a frame rate reduction by N) . 
However, the invention can be extended to other input 
5 and output formats and resolutions as well. 

6.1 Progressive Video MV downscaling (luma) 
This problem appears when all four MBs are coded 

as inter, and use frame prediction. Each MV in those 
MBs is downscaled by two in each direction (horizontal 

10 and vertical) to determine the MV of four blocks in 

MPEG-4 (MPEG-4 allows one MV per 8x8 block) . The 
scaled MVs are then predictively encoded (using a 
median filter) using the normal MPEG-4 procedure: 

Note that each MB (comprising four blocks) has to 

15 be coded in the same mode in both MPEG-2 and MPEG-4. 

With video downscaling, the output MB (four blocks) 
corresponds to four input MBs. 

6.2 Interlaced Video MV downsampling (luma) 

This problem exists when all four MBs are coded as 
20 inter and use field prediction. We need to combine two 

field MVs in each MB to get a frame MV of the resized 
block. Instead of setting the new MV based on the 
spatial activity, the proposed transcoder picks the new 
MV based on its neighbors' MVs. The MVs of all eight 
25 surrounding MBs are used to find a predictor (field MVs 

are averaged in case of MB with field prediction) . The 
median value from these eight MVs becomes a predictor , 
and the field MV of the current MB, which is closer in 
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terms of Euclidean distance, is scaled by two in the 
horizontal direction to become the new MV. 

6.3 MV downsampling (chroma) 

This problem happens when all four MBs are coded 
5 as inter, and use either frame or field prediction 

(MPEG- 4 treats both prediction mode in the same way for 
a chroma block) . The process follows the MPEG-4 method 
to obtain a chroma MV from a luma MV, i.e., a chroma MV 
is the downscaled version of the average of its four 
10 corresponding, 8x8 luma MVs. 

6.4 DCT downsampling (luma progressive, chroma) 
This problem occurs when all four luma MBs are 

coded as intra or inter, and use frame MB structure, 
and their eight chroma blocks (four for Cr and four 
15 for Cb) use either frame or field structure). Dugad's 

method is used to downscale the luma and chroma DCT 
blocks by a factor of two in each direction. 

6.5 Interlaced DCT downsampling (luma) 

This problem arrives in one of two ways. First, 
2 0 its associated MB uses field prediction and second, its 

associated MB uses frame prediction. In either case, 
we want to downscale four 8x8 field DCT blocks (two for 
the top field, and two for the bottom field) into one 
8x8 frame DCT block. The solution for the first case 
25 is to use the same field DCT block as the one chosen 

for MC. The second case involves deinterlacing and we 
propose a combination of the Dugad and Yim methods, 
discussed above. 
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Specifically, the transcoder first downscales four 
field blocks in the vertical direction (and at the same 
time performs deinterlacing) based on the Yim algorithm 
to obtain two frame blocks . The transcoder then 
5 downscales these two frame blocks in the horizontal 

direction to get the output block using the Dugad 
algorithm. 

This is illustrated in FIG. 6, where four 8x8 
coefficient field-mode DCT blocks are shown at 600, two 
10 8x8 frame-mode DCT blocks are shown at 610, and one 8x8 

frame-mode DCT block is shown at 620. 

The procedure for DCT downscaling in accordance 
with the invention can be summarized as follows: 

1. Form the 16x16 coefficient input matrix by 

15 combining four field blocks together as shown at 600. 

2. For vertical downscaling and filtering, apply a 
low pass (LP) filter D according to Yim's algorithm to 
every row of the input matrix. The LP input matrix is 
now 16x8 pixels, as shown at 610. 

20 3. Form B x and B 2 8x8 matrices from the LP matrix 

4. Perform a horizontal downscaling operation 
according to Dugad' s algorithm to every column of B 2 
and B 2 to obtain the output matrix (8x8) (620) as 
25 follows: 

B = B^li? 4- B 2 (T R Tl? = (2?! 4- B 2 )E + (5, - B 2 )0 
where E and O denote even and odd rows as 
discussed above. 



WO 01/95633 



PCT/US01/40811 



42 



In particular, a horizontal downs amp ling matrix 
composed of odd "0" and even "E" matrices as follows 
may be used (ignoring the scaling factor) : 





E = 


[e(0) 
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0 


o, 
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coefficients as 


follows i 


can be used: 


20 


e(0) 


= 4 






o(0) 


= 2 


.56915448 




e(l) 


= 0. 
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o(l) 


= -0 


.149315668 




e(2) 
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= 2 
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o(3) 
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e(5) 


= 2 






o(5) 


= 0 
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e(6) 
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o(6) 


= 1 
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e(8) = 0.906127446 o(8) =-0.509795579 
e(9) = 1.731445835 o(9) =-0.750660555. 

Essentially, the product of a DCT matrix which is 
sparse is used as the downsampling matrix. 
5 The technique may be extended generally for 2:1 

downsizing of an NxN block that comprises four N/2xN/2 
coefficient field-mode blocks. Other downsizing ratios 
may also be accommodated. 
6.6 Special cases 

10 Special cases occur when all four MBs are not 

coded in the same mode (not falling in any of the five 
previous cases) . We always assume that any intra or 
skipped MB among the other inter MBs are inter mode 
with zero MV. Field MVs are merged based on section 

15 6.2 to obtain frame MV, and then we apply the 

techniques of section 6.1. MC is recommended to 
determine the texture of the intra block, which is 
treated as an inter block with a zero MV by the 
transcoder . 

20 7. Conclusion 

It should now be appreciated that the present 
invention provides a transcoder architecture that 
provides the lowest possible complexity with a small 
error-. This error is generated in the MPEG-4 texture 

25 encoding process (QP coding, DC prediction, nonlinear 

DC scaler) . These processes should be removed in the 
future profile of MPEG-4 to create a near-lossless 
transcoding system. 
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The invention also provides complete details of a 
size transcoder to convert a bitstream of ITU-R 601 
interlaced video coding with MPEG- 2 MP@ ML into a 
simple profile MPEG-4 bitstream which contains SIF 
progressive video suitable for a streaming video 
application. 

For spatial downscaling of field-mode DCT blocks, 
it is proposed to combine vertical and horizontal 
downscaling techniques in a novel manner such that 
sparse downsampling matrixes are used in both the 
vertical and horizontal direction, thereby reducing 
computations of the transcoder . 

Moreover, for MV downscaling, we propose using a 
median value from its eight neighboring MV. This 
proposal works better than algorithms in section 5.2 
since our predicted MV go with the global MV. It also 
works well with an interlaced MB, which has only two 
MVs instead of 4 MVs per MB. 

Although the invention has been described in 
connection with various preferred embodiments, it 
should be appreciated that various modifications and 
adaptations may be made thereto without departing from 
the scope of the invention as set forth in the claims. 
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What is claimed is: 

1. A method for transcoding a pre-compressed 
input bitstream that is provided in a first video 
coding format, comprising the steps of: 

recovering header information of the input 
bitstream; 

providing corresponding header information in a 
second, different video coding format; 

partially decompressing the input bitstream to 
provide partially decompressed data; and 

re-compressing the partially decompressed data in 
accordance with the header information in the second 
format to provide an output bitstream. 

2. The method of claim 1, wherein: 

the first and second video coding formats comprise 
an MPEG-2 format and an MPEG-4 format, respectively. 

3. The method of claim 1, wherein: 

the first video coding format comprises MPEG-2 
Main Profile at Main Level; and 

the second video coding format comprises a simple 
profile MPEG-4 bitstream with Standard Intermediate 
Format (SIF) progressive video. 

4. The method of claim 1, wherein: 
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the partially decompressed data comprises motion 
vectors and Discrete Cosine Transform (DCT) 
coefficients; and 

the second format comprises at least one of a new 
mode decision, AC/DC prediction, and motion 
compensation. 

5. The method of claim 1, wherein: 

at least one look-up table is used to provide the 
corresponding header information in the second video 
coding format . 

6. The method of claim 1, wherein: 
downscaling is performed on the partially 

decompressed data by downsampling DCT coefficients and 
motion vector data thereof* 

7 ♦ The method of claim 1, wherein: 

2:1 downscaling is performed on at least one group 
of four field-mode Discrete Cosine Transform (DCT) 
blocks of the partially decompressed data by performing 
vertical downsampling and de- interlacing thereto to 
obtain a corresponding group of two frame-mode DCT 
blocks, and performing horizontal downsampling to the 
two frame-mode DCT blocks to obtain one frame-mode DCT 
block. 

8. The method of claim 7, wherein: 
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the vertical downsampling also achieves low pass 
filtering of the four field-mode DCT blocks. 

9. The method of claim 7, wherein: 

the vertical and horizontal downsampling use 
respective sparse matrixes. 

10. The method of claim 1, wherein: 

in the recompressing step, a code (DQUANT) which 
specifies a change in a quantizer is set according to a 
differential value of a quantization parameter of the 
partially decompressed data. 

11. The method of claim 1, wherein: 

for. re -compressing intra coded mac rob locks, a 
coded block pattern (CBP) is set according to a 
corresponding value of the partially decompressed data. 

12. The method of claim 1, wherein: 

for re -compressing non- intra coded macroblocks, 
skipped macroblocks in the partially decompressed data 
are coded as not__coded macroblocks, where all Discrete 
Cosine Transform (DCT) coefficients have a zero value. 

13. The method of claim 1, wherein: 

in the recompressing step, predicted motion 
vectors in the partially decompressed data are reset 
according to the second format. 
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14. The method of claim 1, wherein: 

in the recompressing step, dual prime mode 

macroblocks of the partially decompressed data are 

converted into field-coded macroblocks. 

15 • A method for performing 2:1 downscaling on 
video data, comprising the steps of: 

forming at least one input matrix of NxN Discrete 
Cosine Transform (DCT) coefficients from the video data 
by combining four N/2xN/2 field-mode DCT blocks; 

performing vertical downsampling and de- 
interlacing to the input matrix to obtain two N/2xN/2 
frame-mode DCT blocks; 

forming an NxN/2 input matrix from the two frame - 
mode DCT blocks; and 

performing horizontal downsampling to the NxN/ 2 
matrix to obtain one N/2xN/2 frame-mode DCT block. 

16. The method of claim 15, wherein N=16. 

17. The method of claim 15, wherein: 

the vertical downsampling also achieves low pass 
filtering of the NxN input matrix. 

18. The method of claim 15, wherein: 
the vertical downsampling uses a sparse 

downsampling matrix. 

19. The method of claim 18, wherein: 
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the sparse downsampling matrix=0 . 5 [I 8 la], where I 8 
is an 8x8 identity matrix. 

20. The method of claim 15, wherein: 

the horizontal downsampling uses a sparse 
downsampling matrix composed of odd "O" and even "E" 
matrices . 

21. The method of claim 20, wherein: 
the even matrix has the following form: 
E = [e(0) 0 0 0, 

0 e(l) 0 e(2), 
0 0 0 0, 
0 e(3) 0 e(4), 
0 0 e(5) 0, 
0 e(6) 0 e(7), 
0 0 0 0, 
0 e(8) 0 e(9)] 
where e(l) through e(9) are non-zero coefficients; 

and 

the odd matrix has the following form: 
0 = [0 0 0 0, 

o(0) 0 o(l) 0, 

0 o(2) 0 0, 

o(3) 0 o(4) 0, 

0 0 0 0, 

o(5) 0 o(6) 0, 

0 0 0 o(7), 

o(8) 0 o(9) 0] 
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where o(l) through o(9) are non-zero coefficients. 

22. An apparatus for transcoding a pre-compressed 
input bitstream that is provided in a first video 
coding format, comprising: 

means for recovering header information of the 
input bitstream; 

means for providing corresponding header 
information in a second, different video coding format; 

means for partially decompressing the input 
bitstream to provide partially decompressed data; and 

means for re-compressing the partially 
decompressed data in accordance with the header 
information in the second format to provide an output 
bitstream. 

23. An apparatus for performing 2:1 downscaling 
on video data, comprising: 

means for forming at least one input matrix of NxN 
Discrete Cosine Transform (DCT) coefficients from the 
video data by combining four N/2xN/2 field-mode DCT 
blocks ; 

means for performing vertical downsampling and de- 
interlacing to the input matrix to obtain two N/2xM/2 
frame-mode DCT blocks; 

means for forming an NxN/2 input matrix from the 
two frame-mode DCT blocks; and 
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means for performing horizontal downsampling to 
the NxN/2 matrix to obtain one N/2xN/2 frame-mode DCT 
block. 
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